Adaptive SVRG Methods under Error Bound Conditions with Unknown Growth Parameter
نویسندگان
چکیده
Error bound, an inherent property of an optimization problem, has recently revived in the development of algorithms with improved global convergence without strong convexity. The most studied error bound is the quadratic error bound, which generalizes strong convexity and is satisfied by a large family of machine learning problems. Quadratic error bound have been leveraged to achieve linear convergence in many first-order methods including the stochastic variance reduced gradient (SVRG) method, which is one of the most important stochastic optimization methods in machine learning. However, the studies along this direction face the critical issue that the algorithms must depend on an unknown growth parameter (a generalization of strong convexity modulus) in the error bound. This parameter is difficult to estimate exactly and the algorithms choosing this parameter heuristically do not have theoretical convergence guarantee. To address this issue, we propose novel SVRG methods that automatically search for this unknown parameter on the fly of optimization while still obtain almost the same convergence rate as when this parameter is known. We also analyze the convergence property of SVRG methods under Hölderian error bound, which generalizes the quadratic error bound.
منابع مشابه
Supplementary Materials “Adaptive SVRG Methods under Error Bound Conditions with Unknown Growth Parameter”
متن کامل
A Robust Distributed Estimation Algorithm under Alpha-Stable Noise Condition
Robust adaptive estimation of unknown parameter has been an important issue in recent years for reliable operation in the distributed networks. The conventional adaptive estimation algorithms that rely on mean square error (MSE) criterion exhibit good performance in the presence of Gaussian noise, but their performance drastically decreases under impulsive noise. In this paper, we propose a rob...
متن کاملTarget Tracking with Unknown Maneuvers Using Adaptive Parameter Estimation in Wireless Sensor Networks
Abstract- Tracking a target which is sensed by a collection of randomly deployed, limited-capacity, and short-ranged sensors is a tricky problem and, yet applicable to the empirical world. In this paper, this challenge has been addressed a by introducing a nested algorithm to track a maneuvering target entering the sensor field. In the proposed nested algorithm, different modules are to fulfill...
متن کاملADAPTIVE FUZZY TRACKING CONTROL FOR A CLASS OF PERTURBED NONLINEARLY PARAMETERIZED SYSTEMS USING MINIMAL LEARNING PARAMETERS ALGORITHM
In this paper, an adaptive fuzzy tracking control approach is proposed for a class of single-inputsingle-output (SISO) nonlinear systems in which the unknown continuous functions may be nonlinearlyparameterized. During the controller design procedure, the fuzzy logic systems (FLS) in Mamdani type are applied to approximate the unknown continuous functions, and then, based on the minimal learnin...
متن کاملOptimal adaptive leader-follower consensus of linear multi-agent systems: Known and unknown dynamics
In this paper, the optimal adaptive leader-follower consensus of linear continuous time multi-agent systems is considered. The error dynamics of each player depends on its neighbors’ information. Detailed analysis of online optimal leader-follower consensus under known and unknown dynamics is presented. The introduced reinforcement learning-based algorithms learn online the approximate solution...
متن کامل